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Abstract: As the architect of the oldest and most stable performance-funding program, Tennessee 
provides a unique opportunity to analyze the impact of changes in performance-funding policies on 
changes in institutional retention and six-year graduation rates over time. Utilizing spline linear 
mixed models, this study examines the impact of changes in Tennessee’s performance-funding 
policies on retention and six-year graduation rates at public four-year institutions from 1995-2009. 
The results show tying retention and graduation rates to performance-funding was unrelated to 
changes in the performance measures over the fifteen year period examined. Additionally, the 
doubling of the monetary incentive associated with the retention and six-year graduation rate 
measures in 2005 was not associated with increases in retention rates. These results suggest that at 
their current funding levels, states’ adoption of performance-funding programs, such as the one in 
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Tennessee, may be insufficient to incentivize changes in institutional outcomes as desired by state 
leaders. 

Keywords: Performance-funding; accountability; graduation; retention 


Impacto del financiamiento en el rendimiento sobre las tasas de retencion y de graduacion 
Resumen: Como arquitecto del programa de financiamiento por rendimiento mas antiguo y estable, 
el estado de Tennessee provee una oportunidad unica para analizar el impacto en el tiempo de los 
cambios en las politicas de financiamiento por rendimiento sobre los cambios en la retencion 
institucional y las tasas de graduacion en un periodo de seis anos. Utilizando modelos lineales mixtos 
spline , en este estudio se examina el impacto de los cambios en las politicas de financiamiento al 
rendimiento del estado de Tennessee sobre las tasas de retencion y de graduacion en seis anos en 
instituciones publicas con cursos de cuatro anos de duration, desde 1995 hasta 2009. Los resultados 
muestran que atar las tasas de retencion y de graduacion al financiamiento del rendimiento no tiene 
relation con los cambios en las medidas de rendimiento en el periodo de 15 anos examinado. 
Ademas, la duplication del incentivo monetario asociado con las tasas de retencion y de graduacion 
en seis anos en 2005 no esta asociado con los aumentos en las tasas de retencion. Estos resultados 
sugieren que en los niveles de financiamiento corrientes, la adoption de programas de 
financiamiento al rendimiento por parte de los estados, tal como ocurre en Tennessee, puede ser 
insuficiente para satisfacer el deseo de los lideres estaduales de incentivar cambios en los resultados 
institucionales. 

Palabras clave: financiamiento al rendimiento; rendition de cuentas; graduacion; retencion. 

Impacto no desempenho de financiamento sobre as taxas de retenyao 
Resumo: Como um arquiteto do programa de financiamento por desempenho mais antigo e 
estavel, o estado de Tennessee oferece uma oporunidade unica para analisar o impacto ao longo do 
tempo nas mudanyas nas politicas de financiamento por desempenho sobre as mudanyas 
narentenyao institucional e nas taxas de graduayao em seis anos. Usando modelos linear mistos 
spline este estudo analisa o impacto das mudanyas nas politicas de financiamento por desempenho do 
estado do Tennessee sobre as taxas de retenyao e graduayao em seis anos em instituicoes pupblicas 
com cursos de quarto anos de durayao, no periodo de 1995-2009. Os resultados mostram que 
amarrar taxas de retenyao e graduayao ao financiamento do desempenho nao tern relayao com 
mundanyas nas medidas de desempenho no periodo de 15 anos examinados. Alem disso, a 
duplicayao no incentivo monetario associado com as taxas de retenyao e graduayao em seis anos 
medidas em 2005, nao esta associada a aumentos nas taxas de retenyao. Estes Resultados sugerem 
que nos niveis de financiamento atual, a adoyao de programas de financiamento por desempenho 
por parte dos Estados, como no Tennessee, pode ser insuficiente para safisfazer o desejo de lideres 
do estado para incentivar mudanyas nos resultados institucionais. 

Palavras-chave: desempenho financeiro; responsabilidade; graduayao; retenyao. 


Introduction 

Public higher education institutions continue to find themselves involved in a national 
discourse over concerns of accountability, assessment, and performance. In an attempt to realign 
institutional goals and state priorities, states are tying a portion of state appropriations to 
institutional performance measures such as retention and six-year graduation rates. Beginning with 
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Tennessee in the late 1970s, there has been significant growth in the number of states adopting 
performance-funding programs, reaching a high of 25 states in 2003 (McLendon, Hearn, & Deaton, 
2006). Additionally, a 2009 Midwestern Higher Education Compact (MHEC) report indicates 
several more states (Indiana, Louisiana, and Texas) are considering or have recently adopted 
performance-funding incentives in an attempt to increase the performance of public institutions. 

In addition to the growth in the number of states adopting performance-funding programs, 
states are also increasing the percentage of state appropriations that are tied to performance 
measures. Historically, the pool of state appropriations tied to performance-funding has remained 
around 5% (Petrides, McClelland, & Nodine, 2004); however, by 2014 at least three states, 

Louisiana, Indiana, and Tennessee, will have over 25% of their appropriations to higher education 
tied to outcomes. More than three decades after performance-funding was initially adopted by 
Tennessee, the idea of tying state appropriations to institutional performance remains a “hot topic” 
in state legislatures across the nation (Burke, Modarresi, & Serban, 1999, p. 17). The question 
remains: how effective is performance-funding as a policy tool? 

To better understand the validity of linking state appropriations to institutional performance, 
researchers need to examine stable programs. Doyle and Noland (2006) suggest that, “the increased 
national emphasis on accountability and assessment has placed institutions under the crosshairs of 
legislative oversight. In order to ensure that both internal and external accountability concerns are 
addressed by this movement to direct legislative oversight, it is critical that scholars examine ‘stable’ 
performance-funding programs” (p. 7). 

Burke and Modarresi (2001) selected the performance-funding programs in Tennessee and 
Missouri as representative stable programs based on “age, continuity and acceptance by state and 
campus leaders” (p. 53). Tennessee, which pioneered performance-funding in the 1970s, operates 
the most stable performance-funding program, since it has maintained a continuous program, while 
other states programs, including Missouri (MHEC, 2009), have aborted their programs or are too 
infantile to be effectively evaluated (Burke & Serban, 1999). The growth in states adoption of 
performance-funding policies in an attempt to improve institutional performance begs the question: 
Are performance-funding policies working? 

This study examines the impact of performance-funding on institutional outcomes. 
Specifically, the study analyzes the impact of the adoption and evolution of Tennessee’s 
performance-funding program on Tennessee public four-year institutional outcomes (retention and 
six-year graduation rates) over a fifteen year time period (1995-2009). The stability of the 
performance-funding program in Tennessee provides a unique opportunity to analyze the impact of 
performance-funding on institutional outcomes over an extended time period. 

Review of the Literature 

Historically, the public has had great confidence and trust in public higher education. 
However, it is clear that public land-grant universities’ commitment to the social compact is eroding 
(Couturier, 2005), and as a result, Lingenfleter (2008) notes, “many in the public lack the confidence 
that additional investment will generate the results we need” (p. 4). For instance, U.S. colleges and 
universities are failing to graduate a greater percentage of baccalaureate students than in previous 
decades. While the number of students enrolling in colleges after high school graduation has 
increased from 45.9% in 1974 to 71.5% in 2004 (Horn, Berger, & Caroll, 2004), during the same 
period the six-year graduation rate has lingered around 66% (Adelman, 2006). This means that more 
students are accessing higher education; however, more of them are also leaving without a degree. 
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Public higher education continues to find itself in an “age of disenchantment” (Barnett, 

1992, p. 216), and the public is no longer willing to accept peer-review and accreditation as 
satisfactory forms of accountability. In an attempt to improve institutional outcomes, states have 
altered their accountability strategies over time from system efficiency, to educational quality, to 
organizational productivity, and finally, to external responsiveness to public priorities or market 
demands (Burke, 2005). More recently, states have also tried using performance-funding, 
performance-budgeting and performance-reporting as accountability methods to link state funding 
and priorities to institutional performance. 

Performance-funding links state funding to prescribed performance measures agreed upon 
by the state and institution(s). If the institution meets the measures, it receives a predetermined 
amount of state appropriations (Burke & Minassians, 2003; McLendon, Hearn, & Deaton, 2006). 
Performance-funding provides states with the most direct form of influence on institutional 
outcomes. Performance-budgeting allows state officials complete control to determine if any state 
funding should be appropriated to an institution based upon performance measures (Burke & 
Minassians, 2003; McLendon, Hearn, & Deaton, 2006). This approach allows more flexibility in the 
measures selected and has been more widely implemented; by 2003, 35 states utilized the 
performance-budgeting method. Finally, the majority of states, 42, have employed some version of 
performance-reporting. There are no appropriations tied to performance-reporting; however, this 
method relies on institutions to provide information to the state as publicity (McLendon, Hearn, & 
Deaton, 2006). 

Serban and Burke (1998) point out that states have implemented performance-funding to 
“achieve two major goals: increased accountability and improved institutional performance” (p. 157). 
Additionally, Gaither, Nedwek, and Neal (1994), and Layzell (1999) indicate policymakers have a 
renewed interest in incentivizing institutions to perform well in areas that more closely align with 
current and future state needs. Alexander (2000) notes that as a result, “the entire nature of the 
traditional relationship between government and higher education is in the process of significant 
change in stretching the public dollar to serve more students in attempting to maximize economic 
returns” (p. 413). 

However, these attempts at improving institutional outcomes may have proved ineffective 
because, the percentage of state appropriations attached to the performance measures is around 5% 
of the total appropriations to institutions (Petrides, McClelland, & Nodine, 2004). For example, in 
2008 the University of Tennessee at Knoxville received $11.8 million from performance-funding 
(Tennessee Higher Education Commission, 2011), while they received double that amount ($23.6 
million) in private gifts for the same year (U.S. Department of Education, 2008). Additionally, in 
2008 the University of Tennessee at Knoxville’s unrestricted state appropriations represented 54% 
of the institution’s operating revenues (U.S. Department of Education, 2008), while revenue from 
performance-funding only represented 1.5%. 

One exception to the typical performance measure allotment was the state of South 
Carolina, which attempted to link all state funding to performance measures (Alexander, 2000). The 
state abandoned its model, however, after it became clear that the model posed major funding 
deficits for the State’s institutions. Three other states (Arkansas, Colorado, and Minnesota) have also 
abandoned their performance-funding models due to sustainability issues (Burke, Modarresi, & 
Serban, 1999). 

The effectiveness of performance-funding programs continues to receive mixed reviews. On 
the one hand, research indicates there are three main factors contributing to the challenges facing 
implementation of performance-funding: performance indicator selection (e.g., course or degree 
completion, retention, and six-year graduation rates), determination of success (e.g., benchmark) 
criteria, and the limited amount of funding allocated for the policy initiative (Burke & Serban, 1997). 
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Layzell (1999) comments, “the third factor reflects the need to provide a meaningful incentive for 
institutions to take performance-funding seriously. If the dollar value is too low, neither institutions 
nor policymakers will likely find performance-funding worth the effort” (p. 244). In other words, the 
5% allotment may not provide enough of an incentive to comply with the policy, especially if there 
are other (e.g., unrestricted and restricted gifts, grants) revenue pipelines. On the other hand, Burke 
and Modarresi (2001) found performance-funding, in a stable program like Tennessee’s, has 
“improved performance, demonstrated accountability and increased state funding” (p. 445). 

There also appears to be evidence in Tennessee that the longevity of its performance¬ 
funding program and continued support from both state and institutional leaders affirms its success. 
For example, the periodic policy reviews by stakeholders engenders ownership in the continuance of 
the program. Additionally, the performance-funding program, as an accountability policy, continues 
to persist while other states have aborted their programs (Bogue & Johnson, 2010). Furthermore, 
some experts see performance-funding producing remarkable results despite the small percentage of 
a state budget allocated to this kind of program (MHEC, 2009). 

The empirical literature on the impact of performance-funding remains scarce. Most of the 
literature focuses on policy adoption and abandonment. McLendon, Hearn, and Denton (2006) 
examined factors that influenced states to adopt performance-funding policies. The authors found 
that performance-funding and budgeting policies were adopted primarily due to “legislative party 
strength and higher education governance arrangements” (p. 11). More specifically, the authors note 
that states with a greater proportion of Republicans in the state legislature, and those with a more 
decentralized governance structure were more likely to adopt new performance-funding policies. 

Dougherty and Natow (2009) examined the factors that led to performance-funding demise. 
Specifically, the authors examined the termination of performance-funding programs in Florida, 
Illinois, and Washington. Apart from state-specific conditions, the researchers found that: drastic 
reductions in state funding, policy originating from a budget proviso rather than unique legislation, 
lack of strong support from the business community, and a loss of supporters for the policy, are 
common determinants that led to states’ discontinuance of performance-funding. 

A more recent study by Gorbunov (2010) examined states’ adoption, re-adoption, and 
abandonment of performance-funding policies across 47 states for the time period 1979—2009. 
Gorbunov found that the annual change in public enrollment was significant in predicting policy 
adoption. Additionally, the higher the proportion of Republicans in the state legislature, the more 
likely a state was to adopt a performance policy. Coordinating boards with and without budgeting 
authority were also found to have a positive relationship with policy adoption. Policies initiated 
through an appropriation bill or budget proviso were less likely to be readopted than policies 
mandated by legislative act. 

In one of the few studies examining the impact of performance-funding on performance 
measures, Doyle and Noland (2006) examined the relationship between performance-funding and 
retention at Tennessee public institutions. The authors chose to examine Tennessee, because of the 
historical stability of the program. Specifically, the authors’ research focused on changes in retention 
rates within institutions and systems over time. The authors found very few changes in retention 
rates over the period studied associated with performance-funding. They did find, however, that at a 
few institutions performance-funding was associated with increased retention rates. Their research 
suggests mixed results based on the institution; in other words, performance-funding seemed to 
have had a substantive impact on a few institutions, but not the majority. One of the limitations of 
the study is that the authors only examined the within institution variation of retention rates. 
Performance-funding in Tennessee, however, was based on changes in institutional retention and 
graduation rates compared to an institution’s peers (between group variation) as defined in 
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Tennessee’s funding formula (Tennessee Higher Education Commission [THEC], 2000a; THEC, 
2005). 

Shin and Milton (2004) examined whether institutions in states with performance budgeting 
or funding programs experienced improved graduation rates. Utilizing hierarchical linear modeling 
growth analysis, they found that the institutional graduation rates of states with performance¬ 
funding/budgeting did not outperform states without performance-funding programs. The authors 
caution that the lack of growth in graduation rates may not mean that the performance-funding 
policy was ineffective, rather that the five year time period (1997-2001) analyzed may have been 
insufficient for the implementation of these policies to impact graduation rates, a typically slow 
changing measure. Due to this limited time examined in the study, Shin and Milton (2004) go on to 
propose “policy makers are advised to sustain pbf [performance-based funding] programs long 
enough until such programs bear their fruits or prove ineffective” (p.18). 

In summation, much of the literature on performance-funding models has focused on 
stability and successful practices (Wall, Frost, Smith, & Keeling, 2008; Banta, Rudloph, Dyke, & 
Fisher, 1996; Burke, Modarresi, & Serban, 1999; Burke & Modarresi, 2001; Bogue & Johnson, 2010) 
or policy adoption and abandonment (McLendon, Hearn, & Deaton, 2006; Dougherty & Natow, 
2009; Gorbunov, 2010). Additionally, two studies examined the relationship between performance¬ 
funding and student outcomes. Doyle and Noland’s (2006) institutional level study found that 
performance-funding was related to increased student retention rates at a few institutions. While, 
Shin and Milton’s (2004) study showed that institutions in states with performance¬ 
funding/budgeting programs did not outperform institutions in states without performance-based 
programs, over a five-year period. 

A gap in the literature exists in examining the impact these funding models have on student 
performance at public institutions. A review of the existing research demonstrates that the literature 
is divided on the success of states’ adoption of performance-funding programs and on the measures 
these programs are created to incentivize institutions to improve. Regardless of the effectiveness of 
performance-funding, Tennessee continues to implement iterations of its original program and other 
states plan to implement programs in the coming years (MHEC, 2009) or increase the percentage of 
state appropriations tied to performance. 

The existing research has been limited to examining within institution variation (Doyle & 
Noland, 2006) and to examining shorter windows of time, which may have been insufficient for 
measuring the impact of performance-funding policies on institutional outcomes (Shin & Milton, 
2004). This study seeks to fill these gaps by examining the relationship between the adoption of 
performance-funding policies and institutional retention and six-year graduation rates of Tennessee 
public institutions compared to their peers (between group variation) over a fifteen-year time period. 
By examining between group variation, the study’s analyses reflect how institutions were funded 
based on performance-funding in Tennessee. Additionally, analyzing changes in retention and 
graduation rates over a fifteen year period provides policymakers with a longitudinal perspective of 
how these programs are related to changes in institutional outcomes. 

The Evolution of Performance-funding in Tennessee 

Tennessee has long been a leader in the higher education accountability movement, starting 
its performance-funding program in 1979. Since the program’s inception, the goal has been to 
incentivize institutions to improved performance, particularly by enhancing student-learning 
outcomes (Banta et al., 1996). Initially, in 1979-1982 public two- and four-year institutions had the 
opportunity to garner a two percent budget supplement above and beyond the incremental or 
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formula budgeting model as part of the pilot performance-funding program. The budget supplement 
was tied to performance in the following activities: (1) gaining accreditation for qualified programs; 
(2) utilizing standardized examinations to test graduating students in general education and specific 
majors; (3) surveying institutional stakeholders (e.g. currently enrolled students, recent graduates, 
community members and employers) regarding satisfaction with academic programs; (4) conducting 
peer reviews of academic programs; and (5) making campus improvements and academic program 
revisions based on assessment results (Banta et al., 1996). 

While Tennessee is often cited as the most stable performance-funding program, it has 
undergone eight revisions since the initial pilot cycle in 1979 — in 1980, 1982, 1987, 1992, 1997, 

2000, 2005, and 2010. Not only have the criteria (initially internally focused, now allowing external 
peer comparison) evolved over time, but also the budget supplement has increased from 2 to 5.45%. 
Despite these revisions, Tennessee’s program has experienced mild variability compared to many 
states, such as South Carolina. 

In Tennessee’s performance-funding model, retention and six-year graduation rates fell 
under standard three — Student Persistence. The joint six-year graduation and retention rate measure 
was added to the performance-funding model as a factor in the fourth cycle (1997-2000). Initially, 
institutions were compared to a rolling three-year average of their retention and six-year graduation 
rates. The combined rates, as one measure in the performance-funding criteria, were worth 5% of 
the 5% of unrestricted state appropriations tied to performance-funding. Beginning in 2000, 
however, institutions were evaluated on their retention and graduation rate performance relative to 
their funding formula peer institutions (THEC, 2000a). Tennessee’s public institutions’ funding 
formula peers were self-selected by each institution, with the approval of THEC (Russ Deaton, 
Personal Communication, August 2, 2011). In 2005, Cycle 6, the six-year graduation and retention 
measures were divided into separate measures, and the State of Tennessee doubled the monetary 
value of these measures to a combined worth of 10% of the 5.45% allocation. 

Beginning in 2010, Cycle 7, Tennessee has revamped its entire funding model moving from 
an enrollment driven model to an outcomes-based funding model. The new outcomes-based 
funding model funds institutions based on their performance on weighted measures (10 for 4-year 
institutions, 11 for community colleges). The measures account for mission differentiation, so that 
the University of Tennessee at Knoxville’s top measure (6-year graduation rates) is different than 
that of Middle Tennessee State University (number of Bachelors and Associates Degrees). The 
Funding Formula Review Committee, with extensive input from the institutions and systems, 
selected and aligned the goals and priorities for each institution type (as classified by Carnegie) with 
the Complete College Tennessee Act. 

For the first time, 75-80% of state institutions’ entire unrestricted appropriations will be 
allocated based upon outcomes, such as student retention and six-year graduation rates, rather than 
enrollment. The remaining funds are allocated according to the performance-funding program 
(5.45%) and fixed costs. As a result of this new funding formula, retention and six-year graduation 
rates were removed from the performance-funding program and added to the new outcome-based 
funding formula for the appropriate institutions. The performance-funding program has switched its 
focus primarily to quality assurance measures, such as alumni feedback surveys (THEC, 2010). 

Theoretical Framework 

Organizational theory is helpful in understanding why institutions receiving performance¬ 
funding would outperform their peers in measures tied to funding. Resource dependency theory 
(Pfeffer & Salancik, 1978) explains that organizations are inescapably bound to the conditions within 
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their environment. Resources can become more or less scarce, and the organization’s survival is 
contingent on its ability to adapt in response to these environmental changes (Pfeffer & Salanick, 
2003). 

For public institutions, performance-funding represents an opportunity to obtain additional 
funding outside of and at times in addition to the two primary modes of funding: incremental and 
formula (Layzell, 1999). In the performance-funding model, institutions agree with the states on the 
measures tied to their funding. If the performance measures are met, institutions receive the 
funding. As public higher education becomes more market-oriented and as competition over scarce 
resources intensifies, resource dependency theory suggests that institutions would strive to meet the 
measures to ensure their survival and competitiveness. 

The study also adopts Swail’s (2003) Geometric Model of Student Persistence. Swail’s 
model suggests that there are three primary factors that contribute to student persistence: Cognitive, 
Social, and Institutional. For Swail, the institutional level factors consist of “the practices, strategies, 
and culture of a college or university that, in either an intended way or unintended way, impact 
student persistence” (p. 77). Swail’s model recognizes the role and responsibility of the institution in 
helping students to persist and succeed, both academically and socially. In the Geometric model, 
institutional factors make up the base of the model because, for Swail, the institution forms the basis 
for student success. The institution can influence student persistence by providing academic, social, 
and financial support. 

The Geometric model accurately reflects the state’s belief, as evidenced through their 
accountability and performance-funding initiatives, that the institution plays a vital role in the 
success of its student population. Increasingly, states are attempting to incentivize institutions to 
shift their practices, strategies, and culture levers through performance-funding. Institutions that 
improve retention and six-year graduation rates have the opportunity to secure scarce resources in 
an increasingly competitive environment. 

Purpose of Study and Research Questions 

This study attempts to examine the impact of the adoption of performance-funding policies 
on retention and six-year graduation rates at public four-year institutions. Specifically, this study is 
concerned with the impact that the adoption of state performance-funding in Tennessee has had on 
the state’s public four-year institutions’ retention and six-year graduation rates compared to their 
peer institutions. Tennessee’s stability provides the researchers with a unique opportunity to examine 
the impact of the state’s adoption of multiple performance-funding policies on retention and six- 
year graduation rates over time. This study addresses the following research questions: 

1) What is the impact of Tennessee’s adoption of retention rates as a performance measure in 
the state’s performance-funding program in 1997 on institutional retention rates at 
Tennessee public four-year institutions compared to their peers? 

2) What is the impact of Tennessee’s adoption of six-year graduation rates as a performance 
measure in the state’s performance-funding program in 1997 on institutional six-year 
graduation rates at Tennessee public four-year institutions compared to their peers? 

3) What is the impact of Tennessee’s doubling the monetary value associated with its retention 
rate measure in the performance-funding program in 2005 on institutional retention rates at 
Tennessee public four-year institutions compared to their peers? 
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Research Design and Methodology 

The stability of the performance-funding program in Tennessee provides a unique 
opportunity to analyze the impact of these policies on changes in institutional retention and six-year 
graduation rates over time. Operating the oldest and most stable performance-funding program, 
Tennessee serves as a good quasi-experiment in which the outcomes of Tennessee public four-year 
institutions can be compared with their out-of-state peer institutions. The peer institutions used in 
the study were the same peer institutions identified for each institution for comparison in the 
performance-funding formula and are presented in Table A-l (THEC, 2000b). As was previously 
discussed, an institution’s funding formula peers were self-selected by the institution with the 
approval from THEC. 

Of the peer institutions included in the analyses, South Carolina was the only state that had 
institutions engaged in performance-funding during the study’s time frame. The researchers chose to 
keep the three peer institutions from South Carolina in the analysis because they represented only 
three institutions out of the sixty-seven in the peer group, and graphical and descriptive analyses 
suggested that they did not behave as outliers, which could bias the peer group. Rather, they 
behaved very similarly to the rest of the peer institutions. Additionally, evidence suggests that the 
performance-funding experiment in South Carolina had no significant effect on the outcomes of 
institutions (Schmidt, 2001). 

Quasi-experimental designs allow for effects related to the research question of interest to be 
analyzed despite the lack of random assignment (Pedhazur & Schmelken, 1991). To discern the 
response of Tennessee institutions to the performance-funding policies, the analysis utilizes data 
from several sources, which are presented in Table A-2, from 1995-2009. The fifteen years included 
in the analysis allow the researcher to examine the association of the initial adoption of six-year 
graduation and retention rates as a joint measure into the performance-funding formula in 1997 and 
to assess the impact of the state doubling the financial incentive for institutions to increase their six- 
year graduation and retention rate measures in 2005. 

To answer the proposed questions, the study utilizes spline linear-mixed models (LMM) to 
examine the impact of performance-funding policies on institutional six-year graduation rates at 
Tennessee public four-year institutions and their peer institutions. LMMs are the most appropriate 
statistical technique for several reasons. First, using LMMs allows the researchers to ask questions 
about and model within-group change and between-group change simultaneously (Raudenbush & 
Bryk, 2002). Second, the LMM accounts for the nested nature of the data (dependency among the 
repeated measures) through the variance-covariance matrix in the model (Fitzmaurice, Laird, & 
Ware, 2004; Singer & Willett, 2003). Third, LMMs incorporate a random error term that accounts 
for measurement unreliability. Fourth, LMMs account for between group differences by 
incorporating fixed and random effects, which allows the researcher to fit the most parsimonious 
model. Finally, LMMs can also accommodate both dynamic and static predictors. 
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Spline models are appropriate for modeling non-linear trends. Conceptually, splines divide the 
time series axis into segments with differing slopes, while still considering a model for the trend over 
time that is composed of “piecewise linear trends, having different slopes within each segment but 
joined or tied together at fixed times” (Fitzmaurice, Laird, & Ware, 2004, p. 8). The locations where 
the segments meet are referred to as knots. Utilizing splines allows the mean response to vary as 
time proceeds based on the individual regression slopes for each segment. The simplest linear spline 
model has the following form in Linear Mixed Model (LMM) notation: 

(1) E(Yy) = ft l + ft 2 t ij + ft 3 (t iJ -t*) + , 

Where t is the knot, and 

0 ^ if t^t 

tij-t if t V] > t 


In the current study, the use of spline models allows the researchers to compare the mean 
change in retention and six-year graduation rates before the knot, which denotes the performance¬ 
funding policy change event, with the mean change after the knot. To test for this response, the 
following model was specified and is presented in LLM notation: 

Y y = />, + I3 2 ty + ft, (L - 1*) + + /^Performance Funding Group,. 

(2) +/1-A, + /1 6 Performance Funding Group,, x t tj + ft 1 X i x t, 

+/^Performance Funding Group,, x (t - t*) + + (b u + b 2j ty+ e tj ) 

Where / represents the time metric (year), performance-funding group represents whether an 
institution was in Tennessee (engaged in performance-funding) or a peer institution, and X 
represents a matrix of the control variables. In terms of performance-funding group comparison, the 
null hypothesis of no group differences in retention or six-year graduation rates after the performing 
funding policy was implemented is: 

H 0 :ft 6 =ft s =0. 

Organizational theory informs us that organizations adapt to change incrementally and would 
not be able to adjust to these changes in their strategy instantaneously (Perdu & Levis, 1998). To 
capture the effect of the pokey changes and akow institutions incremental time to adjust, three 
separate analyses are conducted. Model A corresponds to research question 1, Model B is the fitted 
model addressing research question 2, and Model C corresponds to research question 3. Each 
analysis akowed four years of observation both before and after the pokeies were introduced. As a 
result, the spkne knot for each analysis is modeled two years after the pokey was introduced, with 
Model C being the exception. 

The first analysis examines the initial pokey adoption, which added retention rates to the 
performance-funding model in 1997 (Model A). The analysis compares the retention rates at 
Tennessee pubkc four-year institutions and their peers from 1995-2003. By choosing the period of 
1995-2003, the study models two years before the pokey was adopted (1995 - 1996) and two years of 
lag time for institutions to respond to the pokey (1997 - 1998). The knot for Model A is placed in 
1999, akowing us to compare the slope of retention rates before institutions were able to influence 
their outcomes (1995-1998) with the years fokowing the pokey introduction (2000-2003). 

The second analysis (Model B) examines the impact of the adoption of six-year graduation 
rates as a performance-funding measure in 1997 by comparing Tennessee pubkc four-year 
institutions and their peers from 1998-2006. The time span akows for the comparison, before and 
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after the policy adoption, of four graduation cohorts. The knot for Model B was placed in 2002. 

Finally, in order to examine the impact of the State’s decision to double the money tied to 
retention and six-year graduation rates, the study conducts a third analysis that models the change in 
retention rates at Tennessee public four-year institutions and their peers from 2001-2009 (Model C). 
Because 2009 is the latest year of available data, the spline knot for Model C was placed at 2005, the 
same year the policy was introduced to allow for four years of observation after the policy was 
implemented. The impact of the State’s increased financial incentive on six-year graduation rates 
could not be examined, because the policy adoption occurred in 2005 and the first six-year cohort 
will not graduate until 2011. 

In addition to the random effects, which account for institutional differences, the study also 
includes additional controls to prevent the results from being driven by other institutional factors 
related to retention and six-year graduation rates (Fitzmaurice, Laird, & Ware, 2004). First, the 
student body characteristics of an institution are likely to impact six-year graduation and retention 
rates. For example, research has shown that there is a positive relationship between a student’s 
socio-economic status and degree completion (Adelman, 1999, 2006; Cabrera, Burkum, & Nasa, 
2005; Titus, 2006a, 2006b). Furthermore, students’ academic preparedness has been shown to be 
related to degree completion (Adelman, 2004; DesJardins, Ahlburg, & McCall, 2002). 1 

Additionally, there is a gap in the graduation rates of minority students compared to white 
students (Adelman, 2004; Horn, 2006; Titus, 2006a, 2006b; DesJardins, et. al., 2002). Adelman 
(2006) and Kim (2007) found that part-time attendance is negatively related to degree completion. 
To account for these factors, the analyses control for the log of total enrollment, the percent of the 
student body that received Pell grants, the percent of the student population that is part-time, and 
the percent of the population that is minority. The financial makeup of an institution also impacts 
six-year graduation and retention rates. For instance, the percentage of revenue from tuition is 
positively related to degree completion (Kim, 2007; Titus, 2006a). Additionally, Titus (2006a) found 
that the level of institutional expenditures on educational expenditures is positively associated with 
degree completion. For this reason, the study also accounts for the percentage of revenue from 
tuition, and the percent of expenditures that an institution spends directly on education. Finally, the 
study also controls for Carnegie classification in order to account for mission differentiation. The 
dependent variables included in the analyses are institutional retention (Model A and Model C) and 
six-year graduation (Model B) rates. 


Results 

Initially, graphical and descriptive analyses were performed to explore patterns in the data. 
Table A-3 presents the descriptive statistics for the variables included in the analyses. The analyses 
and results are presented and discussed by research question. 

(Model A) Research Question 1: What is the impact of Tennessee’s adoption of retention rates as a performance 
measure in the state’s performance funding program in 1997 on institutional retention rates at Tennessee public four- 
year institutions compared to their peers? 


1 Due to limitations in data availability, a proxy for academic preparedness was not included in the analysis 
(no publicly available dataset has an appropriate measure for all institutions included in the sample for the 
fifteen year period examined). 
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In 1995 Tennessee institutions had a mean retention rate of 0.69 (SD = 0.05), by 2003 their 
mean retention rate was 0.68 (SD = 0.07). The average retention rate had declined slightly over the 
period. A paired samples t-test was used to test for a statistically significant difference between the 
retention mean at the beginning of the timeframe compared to the mean in 2003. The results of the 
test, /(8) = 0.257, p = 0.80, suggests that there is no difference in the change of retention rates for 
Tennessee institutions over the time period. Similar results were found for Tennessee peer 
institutions, /(43) = 1.13, p = 0.26, in 1995 (M =0.75, SD =0.07) and in 2003 (M =0.73, SD = 0.12). 
Finally, the two groups’ mean averages were compared at both time points to see if there was a 
statistically significant mean difference in retention scores at the beginning and end points of the 
analysis. The independent samples t-test revealed an intercept effect, /(68) = 2.42, p < 0.018, for 
mean retention rate by performance-funding group. Specifically, Tennessee public institutions’ mean 
retention rate was lower than their peers in the initial year of the study. However, there was no 
statistically significant difference in the mean retention rate in 2003, /(59) = 0.290, p =0.773). 

These results suggest that there may be no effect of performance-funding on retention rates 
over the period studied. To better understand the impact of the adoption of the performance¬ 
funding policy in 1997 on retention rates, a spline linear mixed model was utilized to test for an 
interaction effect between performance-funding group and the mean retention rate change before 
and after the policy adoption. Table 1 presents the results of the fitted model. The model shows that 
performance-funding did not account for statistically significant variation in retention rates in the 
beginning year of the study, /(537) = -1.51, p = 0.13, nor was it associated with change over time, 
/(537) = 1.82, p = 0.06. Institutional revenue was the lone covariate that accounted for significant 
variation in intercepts, /(537) = 3.72, p < 0.05, with higher revenue associated with higher intercepts. 
Lower Carnegie classifications, /(537) = -2.04, p < 0.05, the percent of part-time students, /(537) = - 
2.43, p < 0.05, and the percent of Pell students, /(537) = -3.23, p < 0.05, are all negatively associated 
with changes in retention rate over time. Performance-funding group membership was not 
statistically related with differences in intercepts or retention rate change over time. 

Table 2 presents the results of the test of the hypothesis of interest, where there are no 
group differences in change in retention rates after the performance-funding policy was 
implemented: H 0 : /3 6 = yS 8 = 0 , under Model A. The test of the hypothesis reveals that the two 
parameters of interest are not statistically different from zero, F(2, 527 ) = 0.53, p = 0.22. This 
evidence is further corroborated by the model comparisons presented in Table 3 under Model A. 
The goodness of fit statistics reveal that the reduced model fits the data better than the full model 
(x 2 (2) = 1 -04, p = 0.22). The results show that the implementation of performance-funding was not 
associated with statistically different changes in retention rates at Tennessee institutions compared to 
their peers. 
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Table 1 

Model A Results (N = 76) 


Fixed Effects Variable 

Coef. Std. Error 

t-value 

dP 

Sig 

(Intercept) 

-1.346 

0.511 

-2.63 

537 

** 

Time 

0.168 

0.092 

1.82 

537 


Spline 

-0.006 

0.004 

-1.51 

537 


Performance-funding 

0.013 

0.034 

0.38 

537 


Enrollment (natural log) 

-0.078 

0.043 

-1.82 

537 


Revenue (natural log) 

0.154 

0.041 

3.72 

537 

*** 

Carnegie 

0.025 

0.016 

1.59 

537 


Percent Expenditures on Education 

-0.192 

0.137 

-1.40 

537 


Percent Pell Students 

0.105 

0.076 

1.39 

537 


Percent Minority Students 

0.044 

0.038 

1.17 

537 


Percent Part-time Students 

0.009 

0.084 

0.11 

537 


Enrollment (natural log) x Time 

0.001 

0.008 

0.11 

537 


Revenue (natural log) x Time 

-0.009 

0.008 

-1.18 

537 


Carnegie x Time 

-0.006 

0.003 

-2.05 

537 

* 

Percent Expenditures on Education x Time 

0.042 

0.024 

1.71 

537 


Percent Pell Students x Time 

-0.051 

0.016 

-3.23 

537 

** 

Percent Minority Students x Time 

0.003 

0.007 

0.41 

537 


Percent Part-time Students x Time 

-0.040 

0.016 

-2.43 

537 

* 

Performance-funding x Spline 

-0.015 

0.009 

-1.72 

537 


Performance-funding x Spline 

0.015 

0.010 

1.50 

537 


Random Effects 1 

Std. Dev. Variance 




bi 

b2 

0.050 

0.007 

0.002 

0.001 





0 Estimated degrees freedom 

* = p-value < .05; ** = p-value < .01 ; *** = p-value <.001 


Table 2 


Conditional Analysis for Retention and Graduation Rates 


Performance-funding Interactions 

Model 

Response 

Omnibus Test 0 

Performance-funding x Time 

Performance-funding x Spline 

Coef.(SE) 

Coef.(SE) 

A 

Retention 

F(2,527)=0.53 

-0.015 

0.015 


Rate 


(0.009) 

(0.010) 

B 

Graduation 

F(2,536)=2.03 

-0.001 

-0.005 


Rate 


(0.003) 

(0.005) 

C 

Retention 

F(2,604) = 1.06 

0.007 

-0.009 


Rate 


(0.005) 

(0.008) 


* = p-value < .05; ** = p-value < .01 ; *** = p-value <.001 
° Estimated degrees of freedom 
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Table 3 


Conditional Analysis for Retention and Graduation Rates using Eog-Eikelihood 


Fixed Effects Parameters 


Full Model 


Reduced Model 



Performance-funding 


Performance-funding 



Enrollment (natural log) 


Enrollment (natural log) 



Revenue (natural log) 


Revenue (natural log) 



Carnegie 


Carnegie 



Percent Expenditures on Education 

Percent Expenditures on Education 



Percent Pell Students 


Percent Pell Students 



Percent Minority Students 


Percent Minority Students 



Percent Part-time Students 


Percent Part-time Students 



Enrollment (natural log) x Time 


Enrollment (natural log) x Time 



Revenue (natural log) x Time 


Revenue (natural log) x Time 



Carnegie x Time 


Carnegie x Time 



Percent Expenditures on Education x Time Percent Expenditures on Education x 

Time 


Percent Pell Students x Time 


Percent Pell Students x Time 



Percent Minority Students x Time 


Percent Minority Students x Time 



Percent Part-time Students x Time 


Percent Part-time Students x Time 



Performance-funding x Time 
Performance-funding x Spline 




Goodness of Fit 

Reduced Model 


Full Model 


Model A 

AIC 


-1107 


-2261 

BIC 


-2155 


-2167 

LogLik 


1152 


1151 

Chi Sq. 




1.041 

df 




2 

Sig. 





Model B 

AIC 


-2631 


-2631 

BIC 


-2527 


-2536 

LogLik 


1338 


1336 

Chi Sq. 




4.09 

df 




2 

Sig. 





Model C 

AIC 


-1935 


-1937 

BIC 


-1833 


-1843 

LogLik 


990 


989 

Chi Sq. 




2.19 

df 




2 

Sig. 






* = p-value < .05; ** = p-value < .01 ; *** = p-value <.001 
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(Model B) Research Question 2\ What is the impact of Tennessee’s adoption of six-year graduation rates as a 
performance measure in the state’s performance-fundingprogram in 1997 on institutional six-year rates at Tennessee 
public four-year institutions compared to their peers? 

Four-year institutions in Tennessee had a mean six-year graduation rate of 0.38 (SD = 0.08), 
compared to their peers, whose average six-year graduation rate was 0.45 (SD = 0.14). By 2007, 
Tennessee four-year institutions’ mean six-year graduation rate had increased to 0.41 (SD = .08), and 
their peers had increased to 0.51 (SD = 0.15). A paired samples t-test was used to test for a 
statistically significant difference between the six-year graduation rate in 1999 and 2007. The results 
of the test, f8) = .785, p = .45, suggest that there is no difference in the change of graduation rates 
for Tennessee institutions over the time period. Tennessee’s peer institutions’ mean difference was 
statistically significant, /(44) = 2.37, p < .05, suggesting that Tennessee’s peer institutions’ mean in 
2007 was significantly higher than in 1999. Finally, the groups were compared for mean differences 
in 1999 and 2007. The results of the independent samples t-test revealed there is no intercept effect, 
/(73) = 1.53, p = 0.129; however, there was a statistically significant difference in the mean six-year 
graduation rates in 2007, /(74) = 2.023, p < .05). 

Unlike the 1997 retention model, the descriptive analysis suggests that there may be an effect 
of performance-funding groups on six-year graduation rates over the period studied. A spline linear 
mixed model was utilized to test for an interaction effect between performance-funding group and 
the mean six-year graduation rate change before and after the policy adoption. The results of the 
fitted model are presented in Table 4. The model shows that there is no significant variation in six- 
year graduation rates over time, /(536) = -0.80, p = 0.42; however there is an intercept effect, /(536) 

= 3.13, p < 0.05. Lower Carnegie classification, /(536) = -4.20, p < 0.05, and higher percentages of 
part-time students /(536) = -3.36, p < 0.05, account for significant variation in intercepts. No 
covariates included in the model accounted for unique variance in slopes. As was the case in the 
1997 retention model, performance-funding group membership did not account for unique variance 
in six-year graduation rates’ intercepts or slopes. 

Table 5 presents the results of the test of the hypothesis of interest of no group differences 
in change in retention rates after the new performance-funding policy was implemented: 

H 0 ; /3 6 = /3 8 = 0 . The test of the hypothesis reveals that the two parameters of interest are not 
statistically different from zero, F( 2, 605) = 1.06, p = 0.34. This evidence is further corroborated by 
the model comparisons presented in Table 3 under Model C. The goodness of fit statistics reveal 
that the reduced model fits the data better than the full model (x 2 (2) = 2.19,p = 0.33). Once again, 
the results show that the implementation of performance-funding was not associated with 
statistically different changes in retention rates at Tennessee institutions compared to their peers. 
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Table 4 

Model B Results (N = 76) 


Fixed 

Effects 

Variable 

Coef. 

Std. Error 

t-value 

dP 

Sig 


(Intercept) 

1.258 

0.402 

3.13 

536 

** 


Time 

-0.035 

0.044 

-0.80 

536 



Spline 

0.000 

0.002 

-0.02 

536 



Performance-funding 

-0.040 

0.043 

-0.94 

536 



Enrollment (natural log) 

0.001 

0.029 

0.03 

536 



Revenue (natural log) 

-0.033 

0.024 

-1.40 

536 



Carnegie 

-0.075 

0.018 

-4.20 

536 

*** 


Percent of Expenditures on Education 

0.011 

0.063 

0.18 

536 



Percent Pell Students 

-0.005 

0.034 

-0.14 

536 



Percent Minority Students 

-0.057 

0.048 

-1.20 

536 



Percent Part-time Students 

-0.206 

0.061 

-3.36 

536 

*** 


Enrollment (natural log) x Time 

0.004 

0.004 

0.98 

536 



Revenue (natural log) x Time 

0.001 

0.003 

0.16 

536 



Carnegie x Time 

0.001 

0.001 

0.55 

536 



Percent of Expenditures on Education x Time 

0.001 

0.010 

0.09 

536 



Percent Pell Students x Time 

-0.002 

0.009 

-0.23 

536 



Percent Minority Students x Time 

-0.003 

0.004 

-0.66 

536 



Percent Part-time Students x Time 

-0.004 

0.008 

-0.53 

536 



Performance-funding x Time 

-0.001 

0.003 

-0.44 

536 



Performance-funding x Spline 

-0.005 

0.005 

-1.17 

536 


Random 


Std. 

Variance 




Effects 


Dev. 





bi 

0.110 

0.0136 





b2 

0.005 

0.0005 





° Estimated degrees freedom 

* = Rvalue < .05; ** = Rvalue < .01 ; *** = p-value <.001 


The results of the test of the hypothesis of interest of no group differences in change in six- 
year graduation rates after the policy was implemented: H 0 : /3 6 = /3 8 = 0 are presented in Table 2 
under Model B. The test of the hypothesis reveals that the two parameters of interest are not 
statistically different from zero, F( 2, 536) = 2.03, p = 0.13. The researchers also confirmed these 
results by conducting model comparisons using Log likelihood. The results of the model 
comparison are presented in Table 3 under Model B. The goodness of fit statistics reveal that the 
reduced model fits the data better than the full model (% 2 (2) = 4.09, p = 0.13). Consistent with the 
1997 retention rate model (Model A), the results show that the implementation of performance¬ 
funding was not associated with statistically different changes in six-year graduation rates at 
Tennessee institutions compared to their peers. 

(Model C) Research Question 3: What is the impact of Tennessee doubling the monetary value associated with its 
retention rate measure in the performance-funding program in 2005 on institutional retention rates at Tennessee public 
four-year institutions compared to their peers? 

Four-year institutions in Tennessee had a mean retention rate of 0.69 (SD = 0.04) in 2001, 
compared to their peers whose average retention rate was 0.78 (SD = 0.08). By 2009, Tennessee 
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four-year institutions’ mean retention rate had increased to 0.72 (SD = .05), and their peers had 
failed to increase (mean = 0.78; SD = 0.08). A paired samples t-test was used to test for a statistically 
significant difference between the mean retention rates for both groups in 2001 and 2009. The 
results of the tests suggest that there are no statistically significant differences in the change of 
retention rates for Tennessee institutions, /(8) = 1.092, p = .30, or their peers, /(37) = 2.97, p < .05, 
over the time period. Additionally, the groups were compared for mean differences in 2001 and 
2009. The results of the independent samples t-test showed that there is an intercept effect, /(73) = 
2.97, p < 0.05, as well as a statistically significant difference in the retention rates in 2009, /(74) = 
2.14, p < .05. 

In order to better model the linear retention rate change over time by performance-funding 
group membership, a spline linear mixed model was utilized. The utilization of a spline linear mixed 
model allows the researcher to test for an interaction effect between performance-funding group 
and the mean retention rate change before and after the policy adoption. Table 5 presents the results 
of the fitted model. The results show that there is no significant variation in intercepts, /(605) = 1.16, 
p = 0.24, or slopes, /(605) = -1.60, p = 0.10, over time. The larger the percent of Pell students, ^(605) 
= -4.21, p < 0.05, and the higher the percentage of part-time students, /(605) = -3.04, p < 0.05, are 
associated with lower retention rate intercepts. The higher the percentage of minority students, 

/(605) = -3.04, p < 0.05, and being a member of the peer group in the performance-funding group 
variable, /(605) = -2.44, p < 0.05 is associated with higher retention rate intercepts. No covariates 
included in the model accounted for unique variance in slopes. While performance-funding group 
membership was significant in explaining intercept differences, group membership did not explain 
any unique variation in retention rate slopes. 


Table 5 

Model C Results (N = 76) 


Fixed Effects Variable 

Coef. 

Std. Error 

t-value 

df° 

Sig 

(Intercept) 

0.3761 

0.3238 

1.16 

605 


Time 

-0.0852 

0.0532 

-1.60 

605 


Spline 

0.0004 

0.0035 

0.10 

605 


Performance- funding 

-0.0445 

0.0183 

-2.44 

605 

* 

Enrollment (natural log) 

0.0112 

0.0255 

0.44 

605 


Revenue (natural log) 

0.0211 

0.0249 

0.85 

605 


Carnegie 

0.0111 

0.0092 

1.21 

605 


Percent Expenditures on Education 

-0.0912 

0.0753 

-1.21 

605 


Percent Pell Students 

-0.3023 

0.0717 

-4.21 

605 

*** 

Percent Minority Students 

0.1266 

0.0285 

4.44 

605 

*** 

Percent Part-time Students 

-0.1663 

0.0548 

-3.04 

605 

** 

Enrollment (natural log) x Time 

-0.0022 

0.0044 

-0.51 

605 


Revenue (natural log) x Time 

0.0057 

0.0043 

1.33 

605 


Carnegie x Time 

-0.0024 

0.0014 

-1.71 

605 


Percent Expenditures on Education x Time 

0.0082 

0.0128 

0.65 

605 


Percent Pell Students x Time 

0.0074 

0.0119 

0.62 

605 


Percent Minority Students x Time 

-0.0119 

0.0041 

-2.94 

605 

** 

Percent Part-time Students x Time 

-0.0123 

0.0099 

-1.24 

605 


Performance-funding x Time 

0.0072 

0.0050 

1.42 

605 


Performance-funding x Spline 

-0.0094 

0.0089 

-1.06 

605 


Random Effects 

Std. Dev. 

Variance 




hi 

0.053 

0.0028 




b 2 

0.007 

0.0004 





° Estimated degrees freedom 

* = p-value < .05; ** = p-value < .01 ; *** = p-value <.001 
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Policy Implications 

This study examined: 1) the impact of Tennessee’s adoption of retention and six-year 
graduation rates into the performance-funding program in 1997, and 2) the impact of the State 
doubling the monetary value of these two measures in the performance-funding model in 2005. 
Treating the adoption of these performance-funding policies in Tennessee as a quasi-experiment, the 
study compared the impact of the policy adoptions on Tennessee institutions’ rate of change in 
retention and six-year graduation rates before the policy was adopted with the rate of change for the 
years following the policy adoption compared to their peer institutions. The results of the analyses 
can be summarized as follows: 

The addition of retention rates into the performance-funding model in 1997 was not 
associated with a significant difference in the rate of change in Tennessee institutions’ retention rates 
compared to their peers. The results were similar for the addition of six-year graduation rates into 
the performance-funding model by the State in 1997. The graduation rate of Tennessee institutions 
did not change at a statistically different rate compared to the pre-policy adoption period when the 
institutions were compared to their peers. Finally, the attempt by the State to further incentivize 
improvement in institutional outcomes by doubling the money tied to the retention and graduation 
rate measures in 2005 was unrelated to the institutions’ change in retention rates. 

Despite these monetary incentives, this study’s findings suggest that the adoption of new 
performance-funding policies has little impact on altering institutional outcomes at their current 
funding levels. Our findings support previous hypotheses that institutions do not have enough 
financial incentive to emphasize improving student outcomes (Petrides, McClelland, & Nodine, 

2004; Burke & Serban, 1999). 

Additionally, public four-year institutions are major organizations, where change occurs 
incrementally and is often costly. Increasing 6-year graduation and retention rate performance takes 
time and considerable resources. Simply put, the small percentage of funds tied to performance¬ 
funding (usually around 5%) may not provide enough financial incentive to influence changes in 
institutional outcomes. These incentives may have been further weakened as state appropriations 
continue to decline as a percentage of institutional budgets over the last three decades (Kane, 

Orszag, & Gunter, 2003; Cheslock & Gianneschi, 2008), leading public institutional leaders to 
comment that their institutions, which were once state-supported, then state-assisted, are now only 
state-located (de Vise, 2011). These comments highlight one of the challenges states face in trying to 
influence institutional outcomes while supplying a smaller proportion of institutional revenues. 

Furthermore, during the sixth cycle of performance-funding in Tennessee (2005-2010), 
institutions, as a whole, averaged receiving over 91% of the potential performance-funding revenue 
they were eligible to receive (THEC, 2011). The results of the current study found that on at least 
two of the measures, however, institutions in Tennessee showed no significant difference compared 
to their peer institutions in the measures’ changes over time. The rate at which Tennessee 
institutions are achieving their potential performance-funding suggests that institutions can achieve 
their performance goals with relative ease. 

Our findings suggest that institutions are able to meet the performance requirement without 
improving their outcomes. At the current funding level, the consequences of failing to improve 
institutional outcomes are low. In order to incentivize improvement in institutional outcomes, 
policymakers are left with two options: 1) consider raising the stakes around performance-funding, 
or 2) pursue new strategies to achieve their goals. 

Regarding the first option, beginning in FY 2012, Tennessee is increasing the proportion of 
state appropriations tied to institutional performance outcomes. Recognizing that higher education 
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is key for economic development and the future success of the state, Tennessee recently passed 
major legislation (the Complete College Tennessee Act of 2010), which focuses on more closely 
tying institutional performance to state priorities. A key piece of the new legislation was the creation 
of a new outcomes-based funding model. No longer will performance measures be tied to just 
5.45% of state appropriations. Under the new model as much as 80% of an institution’s 
appropriations from the state will be tied to these measures. 

While the current study has shown that tying five percent of appropriations to performance 
measures has not been significant in influencing changes in institutional outcomes, the verdict 
remains unsettled on whether Tennessee’s new policy of tying a higher percentage of appropriations 
to these measures will sufficiently incentivize the desired change in institutional outcomes. Future 
research should examine the impact of Tennessee’s new funding policy on institutional performance. 
Additionally, the evolution of the proportion of Tennessee’s state appropriations tied to 
performance measures raises the question: What proportion of appropriations tied to performance 
measures is enough to incentivize changes in institutional outcomes? 

With regard to the second option, while performance-funding policies may be politically 
desirable, our results suggest that policymakers may want to consider utilizing alternative policies to 
achieve improved institutional outcomes. Research on the use of performance-based accountability 
policies and their effectiveness in government has a rich history in the performance management 
literature, which may be informative for the ongoing policy dialogue. Specifically, scholars have 
found little evidence that other performance-based budgeting programs have produced the desired 
results (Andrews & Hill, 2003; Moynihan & Andrews, 2011). Given that scholars have found similar 
results across multiple fields, the effectiveness of these policies to generate the desired results is 
questionable. While the research findings speak to the tangible benefits of these policies, it is 
important not to devalue the importance these policies may hold both politically and culturally. 

Finally, we believe it would be a mistake to conclude from our findings that performance¬ 
funding has not been an effective policy. It may be more appropriate to reframe discussions of its 
effectiveness around implementation and quality assurance rather than performance. The state’s long 
history of performance-funding, which spans over three decades, established a new framework for 
accountability for higher education that focused on institutional performance and outcomes. 
Performance-funding has changed the conversation and culture of expectations of both the public 
and higher education in Tennessee by tying some state appropriations to outcomes. Given that the 
study’s results showed that institutional outcomes did not improve over the period studied 
compared to their peer institutions, the high success rate of Tennessee institutions in receiving their 
performance-funding dollars may raise concerns over how effective the measures are that are being 
used to evaluate institutional performance. Regardless, the state’s long history of tying state 
appropriations to outcomes may have made tying 80% of state appropriations to outcomes more 
politically palatable and culturally feasible. In short, participation in performance-funding created a 
climate that was conducive to expanding the reforms embedded in the Complete College Tennessee 
Act. 


Conclusion 

In an attempt to increase accountability and incentivize institutional performance, numerous 
states have followed Tennessee’s example by establishing performance-funding programs. 
Additionally, at least three states (Indiana, Louisiana, and Texas) have recently adopted or are 
considering adopting performance-funding programs (MHEC, 2009). While the study’s findings are 
obviously particular to Tennessee, the study’s results have implications for state policy makers 
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considering adopting new performance-budgeting programs, higher education governance boards 
that are advising legislative leaders on the merits of these programs, and the public. 

This paper provides evidence that public institutions in Tennessee have not responded to 
the current monetary incentives created by the State’s adoption of performance-funding policies. 

The introduction of retention and six-year graduation rates as a measure included in performance¬ 
funding in 1997 did not result in a statistically significant difference in the mean retention or six-year 
graduation rates at Tennessee institutions compared to their peers. Additionally, the doubling of the 
monetary incentive associated with the retention and six-year graduation rate measures by the State 
in 2005 was not associated with increases in retention rates at Tennessee institutions compared to 
their peer institutions. 

These results suggest that States’ adoption of performance-funding programs, such as the 
one in Tennessee, may not incentivize the change in institutional outcomes as desired by state 
leaders at their current funding levels. Previously, Shin and Milton (2004) recommended that policy 
makers should sustain performance-based funding programs until sufficient time has passed and 
they either result in increased performance or are shown to be ineffective. Based on the study’s 
results, policy makers may want to: 1) consider increasing the financial incentives tied to these 
policies in order to elicit their desired change in institutional outcomes, or 2) consider other methods 
to improve institutional outcomes. 
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Appendix 


Table A-l 

Tennessee Peer Institutions 


University of TN - Chattanooga 

Peers 

Western Carolina University 
Murray State University (KY) 
University of Arkansas Little Rock 
University of West Florida 
Louisiana Tech University 
LTniversity of North Florida 
Sam Houston State LTniversity (TX) 
LTniversity of N. Carolina Wilmington 
Morgan St. LTniversity (MD) 
Appalachian State University (NC) 
Florida A&M LTniversity 
North Carolina A&T 

Austin Peay State University 

Peers 

Jacksonville State University 
Morehead State University 
McNeese State LTniversity 
Salisbury State University 
Valdosta State University 
Sam Houston State University 
Morgan State University 
Texas A&M U. - Corpus Christi 
Appalachian State 
Florida A&M LTniversity 
North Carolina Central University 
North Carolina A&T LTniversity 

Tennessee State University 

Peers 

LTniversity of Arkansas, Fayetteville 
Florida A&M LTniversity 
North Carolina A&T 
South Carolina State LTniversity 
Delaware State University 
Georgia State University 
Northern Kentucky University 
Alcorn State University 
LTniversity of N. Carolina, Charlotte 
Sam Houston State University 
Old Dominion LTniversity 
Virginia State University 


University of TN - Knoxville 

Peers 

LTniversity of Florida 
LTniversity of Georgia 
Texas A&M 

Louisiana State University 
Virginia Polytechnical Institute 
Auburn University 
LTniversity of Kentucky 
North Carolina State University 
LTniversity of Texas at Austin 
University of N. Carolina, Chapel Hill 
LTniversity of Maryland, College Park 
LTniversity of Virginia, Main Campus 

East Tennessee State University 

Peers 

East Carolina University 
The LTniversity of South Alabama 
The LTniv. of Arkansas, Little Rock 
Florida A&M LTniversity 
Florida Atlantic University 
Eastern Kentucky LTniversity 
Appalachian State 
LTniversity of N. Carolina, Charlotte 
Univ. of N. Carolina at Greensboro 
The LTniversity of Texas at Arlington 
The LTniversity of Texas at El Paso 
Old Dominion 

Tenn. Technological University 

Peers 

University of Alabama-Huntsville 
LTniversity of North Florida 
Murray State LTniversity 
Louisiana Tech University 
Morgan State University 
The LTniversity of Texas at El Paso 
The LTniversity of South Alabama 
Western Carolina LTniversity 
Sam Houston State University 
Appalachian State 
Florida A&M LTniversity 
North Carolina A&T University 


University of TN - Martin 

Peers 

Frostburg State University (MD) 
LTniversity of N. Carolina Wilmington 
Salisbury State University (MD) 
Winthrop University (SC) 
Jacksonville State University (AL) 
Radford University 
Arkansas Tech University 
Northern Kentucky University 
Murray State University 
Western Carolina 
Sam Houston State University 
Appalachian State 

Middle Tennessee State University 

Peers 

Florida Atlantic University 
Florida International LTniversity 
LTniversity of Central Florida 
Georgia State University 
LTniversity of Southern Mississippi 
Univ. of N. Carolina at Greensboro 
LTniversity of North Texas 
Old Dominion LTniversity 
Georgia Southern 
LTniversity of New Orleans 
LTniversity of Texas — Arlington 
George Mason LTniversity 

University of Memphis 

Peers 

Georgia State University 
The University of Houston 
LTniversity of Alabama 
LTniversity of Arkansas, Fayetteville 
LTniversity of South Florida 
LTniversity of Louisville 
Florida International LTniversity 
Univ. of Oklahoma Norman Campus 
LTniversity of S. Carolina, Columbia 
Texas Tech University 
George Mason LTniversity 
Virginia Commonwealth LTniversity 
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Table A-2 
Data Sources 


Variable Type 

Variable 

Metric 

Calculated 

Source 

Dependent 

Variables 

Retention 

Rate 

Percent 


Southern Regional 
Education Data 
Board Exchange 


Graduation 

Rate 

Percent 


IPEDS 

Independent 

Variable 

Performance¬ 

funding 

Dummy Code: 

0 = No 

1 = Yes 


THEC SIS 

Control 

Variables 

Total 

Enrollment 

Natural Log of 
Enrollment 


Integrated 
Postsecondary 
Education Data 
System (IPEDS) 


Tuition 

Revenue 

Percent 

Tuition & Fees Revenue 
divided by Total 
Operating Revenue 

IPEDS 


Instructional 

Expenditures 

Percent 

Instruction expenditures 
divided by Total 
Expenditures 

IPEDS 


Carnegie 

Classification 

Dummy Code: 

0 = Research 
Universities 

1 = 

Doctoral/Research 

Universities 

2 = Masters Colleges 
Percent 


Carnegie 

Foundation 


% Minority 

Number of non-white 
students divided by 
number of white 
Students 

IPEDS 


% Receive 
Pell Grant 

Percent 

Number of Students 
who received Pell 
Grants divided by Total 
Enrollment 

Derived Variable 
from IPEDS 


% Part-Time 

Percent 

Number of Part-time 
Students divided by 
Total Enrollment 

Derived Variable 
from IPEDS 
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Table A-3 


Descriptive Statistics 



Variable 

* 

n 

M 

SE 

Mean 

SD 

Min 

Max 


Retention rate 

660 

0.753 

0.004 

0.091 

0.33 

0.97 


Performance-funding 

660 

0.123 

0.013 

0.328 

0.00 

1.00 


% PeU 

648 

0.263 

0.008 

0.200 

0.03 

2.20 


(Log) Revenue 

660 

18.608 

0.033 

0.841 

16.27 

20.38 

Model A 

(Log) Enrollment 

660 

9.544 

0.026 

0.667 

7.95 

10.86 


Carnegie 

660 

1.677 

0.045 

1.149 

0.00 

3.00 


% Educational Experience 

660 

0.456 

0.003 

0.083 

0.20 

0.71 


% Pell 

660 

0.267 

0.004 

0.109 

0.10 

0.59 


% Minority 

660 

0.291 

0.009 

0.252 

0.03 

0.96 


Graduation Rate 

680 

0.477 

0.005 

0.145 

0.02 

0.93 


Performance-funding 

680 

0.119 

0.012 

0.324 

0.00 

1.00 


% Pell 

676 

0.251 

0.005 

0.123 

0.04 

0.88 


(Log) Revenue 

680 

18.811 

0.032 

0.826 

17.16 

20.58 

Model B 

(Log) Enrollment 

680 

9.611 

0.025 

0.661 

7.97 

10.86 


Carnegie 

680 

1.690 

0.044 

1.147 

0.00 

3.00 


% Educational Experience 

680 

0.455 

0.003 

0.085 

0.19 

0.57 


% Pell 

680 

0.252 

0.004 

0.099 

0.09 

0.57 


% Minority 

680 

0.315 

0.010 

0.249 

0.04 

0.97 


Retention rate 

682 

0.767 

0.003 

0.09 

0.19 

0.97 


Performance-funding 

682 

0.119 

0.012 

0.324 

0.00 

1.00 


% PeU 

682 

0.253 

0.005 

0.120 

0.04 

0.74 


(Log) Revenue 

682 

18.895 

0.032 

0.829 

17.16 

20.58 

Model C 

(Log) Enrollment 

682 

9.667 

0.025 

0.662 

8.04 

11.00 


Carnegie 

682 

1.685 

0.044 

1.147 

0.00 

3.00 


% Educational Experience 

682 

0.462 

0.003 

0.084 

0.180 

0.65 


% PeU 

682 

0.241 

0.004 

0.095 

0.080 

0.57 


% Minority 

682 

0.363 

0.011 

0.282 

0.040 

1.00 




Impact of Performance-funding on Retention and Graduation Rates 


27 


About the Authors 

Thomas Sanford 

Tennessee Higher Education Commission 
Email: thomas.sanford@tn.gov 

Thomas Sanford is the Director of Research at the Tennessee Higher Education 
Commission in Nashville, Tennessee. His research examines how colleges’ and 
universities’ interaction with their external environment informs their behavior. He 
holds a Ph.D. from the University of Minnesota. 

James M. Hunter 
University of Minnesota 
Email: hunte218@umn.edu 

James M. Hunter is a doctoral candidate in Educational Policy and Administration at 
the University of Minnesota and the Director of Adult & Graduate Studies at Crown 
College. His research interests focus on the higher education finance, and 
organizational theory. 


education policy analysis archives 

Volume 19 Number 33 30 th of November 2011 ISSN 1068-2341 


© 


SOME RIGHTS RESERVED 


Readers are free to copy, display, and distribute this article, as long as the work is 
attributed to the author(s) and Education Policy Analysis Archives, it is distributed for non¬ 
commercial purposes only, and no alteration or transformation is made in the work. More 
details of this Creative Commons license are available at 

http://creativecommons.Org/licenses/by-nc-sa/3.0/. All other uses must be approved by the 
author(s) or EPAA. EPAA is published by the Mary Lou Fulton Teachers College at Arizona 
State University Articles are indexed EBSCO Education Research Complete, DIALNET, 
Directory of Open Access Journals. ERIC, H.W. WILSON & Co, QUALIS — A 2 (CAPES, Brazil), 
SCOPUS, SOCOLAR-China. 


Please contribute commentaries at http://epaa.info/wordpress/ and send errata notes to 
Gustavo E. Fischman fischman@asu.edu 










Education Policy Analysis Archives Vol. 19 No. 33 


28 


education policy analysis archives 
editorial board 

Editor Gustavo E. Fischman (Arizona State University) 

Associate Editors: David R. Garcia & Jeanne M. Powers (Arizona State University) 


Jessica Allen University of Colorado, Boulder 

Gary Anderson New York University 

Michael W. Apple University of Wisconsin, 
Madison 

Angela Arzubiaga Arizona State University 

David C. Berliner Arizona State University 
Robert Bickel Marshall University 
Henry Braun Boston College 

Eric Cambum University of Wisconsin, Madison 
Wendy C. Chi* University of Colorado, Boulder 
Casey Cobb University of Connecticut 
Arnold Danzig Arizona State University 

Antonia Darder University of Illinois, Urbana- 
Champaign 

Linda Darling-Hammond Stanford University 

Chad d'Entremont Strategies for Children 
John Diamond Harvard University 
Tara Donahue Learning Point Associates 
Sherman Dorn University of South Florida 

Christopher Joseph Frey Bowling Green State 
University 

Melissa Lynn Freeman* Adams State College 
Amy Garrett Dikkers University of Minnesota 
Gene V Glass Arizona State University 
Ronald Glass University of California, Santa Cruz 
Harvey Goldstein Bristol University 
Jacob P. K. Gross Indiana University 

Eric M. Haas WestEd 

Kimberly Joy Howard* University of Southern 
California 

Aimee Howley Ohio University 
Craig Howley Ohio University 
Steve Klees University of Maryland 

Jaekyung Lee SUNY Buffalo 


Christopher Lubienski University of Illinois, 
Urbana-Champaign 

Sarah Lubienski University of Illinois, Urbana- 
Champaign 

Samuel R. Lucas University of California, 

Berkeley 

Maria Martinez-Coslo University of Texas, 
Arlington 

William Mathis University of Colorado, Boulder 
Tristan McCowan Institute of Education, London 

Heinrich Mintrop University of California, 
Berkeley 

Michele S. Moses University of Colorado, Boulder 
Julianne Moss University of Melbourne 
Sharon Nichols University of Texas, San Antonio 
Noga O'Connor University of Iowa 

Joao Paraskveva University of Massachusetts, 
Dartmouth 

Laurence Parker University of Illinois, Urbana- 
Champaign 

Susan L. Robertson Bristol University 

John Rogers University of California, Los Angeles 

A. G. Rud Purdue LTniversity 

Felicia C. Sanders The Pennsylvania State 
University 

Janelle Scott LTniversity of California, Berkeley 

Kimberly Scott Arizona State LTniversity 
Dorothy Shipps Baruch College/CUNY 
Maria Teresa Tatto Michigan State LTniversity 
Larisa Warhol University of Connecticut 
Cally Waite Social Science Research Council 

John Weathers LTniversity of Colorado, Colorado 
Springs 

Kevin Weiner LTniversity of Colorado, Boulder 
Ed Wiley LTniversity of Colorado, Boulder 

Terrence G. Wiley Arizona State LTniversity 
John Willinsky Stanford University 

Kyo Yamashiro LTniversity of California, Los 
Angeles 

* Members of the New Scholars Board 



Impact of Performance-funding on Retention and Graduation Rates 


29 


archivos anali'ticos de politicas educativas 
consejo editorial 

Editor: Gustavo E. Fischman (Arizona State University) 

Editores. Asociados Alejandro Canales (UNAM) y Jesus Romero Morante (Universidad de Cantabria) 


Armando Alcantara Santuario Instituto de 
Investigaciones sobre la Universidad y la 
Educacion, UNAM Mexico 
Claudio Almonacid Universidad Metropolitana de 
Ciencias de la Educacion, Chile 
Pilar Arnaiz Sanchez Universidad de Murcia, 

Espana 

Xavier Besalu Costa Universitat de Girona, Espana 
Jose Joaquin Brunner Universidad Diego Portales, 
Chile 

Damian Canales Sanchez Instituto Nacional para 
la Evaluation de la Educacion, Mexico 
Maria Caridad Garcia Universidad Catolica del 
Norte, Chile 

Raimundo Cuesta Fernandez IES Fray Luis de 
Leon, Espana 

Marco Antonio Delgado Fuentes Uni versidad 
Iberoamericana, Mexico 
Ines Dussel FLACSO, Argentina 

Rafael Feito Alonso Universidad Complutense de 
Madrid, Espana 

Pedro Flores Crespo LTniversidad Iberoamericana, 
Mexico 

Veronica Garcia Martinez LTniversidad Juarez 
Autonoma de Tabasco, Mexico 
Francisco F. Garcia Perez LTniversidad de Sevilla, 
Espana 

Edna Luna Serrano LTniversidad Autonoma de Baja 
California, Mexico 

Alma Maldonado Departamento de Investigaciones 
Educativas, Centro de Investigation y de 
Estudios Avanzados, Mexico 
Alejandro Marquez Jimenez Instituto de 
Investigaciones sobre la Universidad y la 
Educacion, LTNAM Mexico 
Jose Felipe Martinez Fernandez LTniversity of 
California Los Angeles, USA 


Fanni Munoz Pontificia LTniversidad Catolica de 
Peru 

Imanol Ordorika Instituto de Investigaciones 
Economicas — UNAM, Mexico 

Maria Cristina Parra Sandoval LTniversidad de 
Zulia, Venezuela 

Miguel A. Pereyra Universidad de Granada, Espana 

Monica Pini LTniversidad Nacional de San Martin, 
Argentina 

Paula Razquin LTNESCO, Francia 

Ignacio Rivas Flores LTniversidad de Malaga, 

Espana 

Daniel Schugurensky LTniversidad de Toronto- 

Ontario Institute of Studies in Education, Canada 

Orlando Pulido Chaves LTniversidad Pedagogica 
Nacional, Colombia 

Jose Gregorio Rodriguez LTniversidad Nacional de 
Colombia 

Miriam Rodriguez Vargas LTniversidad Autonoma 
de Tamaulipas, Mexico 

Mario Rueda Beltran Instituto de Investigaciones 
sobre la Universidad y la Educacion, UNAM 
Mexico 

Jose Luis San Fabian Maroto LTniversidad de 
Oviedo, Espana 

Yengny Marisol Silva Laya Universidad 
Iberoamericana, Mexico 

Aida Terron Banuelos LTniversidad de Oviedo, 
Espana 

Jurjo Torres Santome LTniversidad de la Coruna, 
Espana 

Antoni Verger Planells LTniversity of Amsterdam, 
Holanda 

Mario Yapu LTniversidad Para la Investigation 
Estrategica, Bolivia 



Education Policy Analysis Archives Vol. 19 No. 33 


30 


arquivos anali'ticos de politicas educativas 
conselho editorial 

Editor: Gustavo E. Fischman (Arizona State University) 
Editores Associados: Rosa Maria Bueno Fisher e Luis A. Gandin 

(Universidade Federal do Rio Grande do Sul) 


Dalila Andrade de Oliveira Universidade Federal de 
Minas Gerais, Brasil 

Paulo Carrano Universidade Federal Fluminense, 
Brasil 

Alicia Maria Catalano de Bonamino Pontificia 
Universidade Catolica-Rio, Brasil 

Fabiana de Amorim Marcello Universidade 
Luterana do Brasil, Canoas, Brasil 

Alexandre Fernandez Vaz Universidade Federal de 
Santa Catarina, Brasil 

Gaudencio Frigotto Universidade do Estado do Rio 
de Janeiro, Brasil 

Alfredo M Gomes Universidade Federal de 
Pernambuco, Brasil 

Petronilha Beatriz Gonsalves e Silva Universidade 
Federal de Sao Carlos, Brasil 

Nadja Herman Pontificia Universidade Catolica- 
Rio Grande do Sul, Brasil 

Jose Machado Pais Instituto de Ciencias Sociais da 
Universidade de Lisboa, Portugal 

Wenceslao Machado de Oliveira Jr. Universidade 
Estadual de Campinas, Brasil 


Jefferson Mainardes Universidade Estadual de 
Ponta Grossa, Brasil 

Luciano Mendes de Faria Filho Universidade 
Federal de Minas Gerais, Brasil 

Lia Raquel Moreira Oliveira Universidade do 
Minho, Portugal 

Belmira Oliveira Bueno Universidade de Sao Paulo, 
Brasil 

Antonio Teodoro Llniversidade Lusofona, Portugal 

Pia L. Wong California State University Sacramento, 
U.S.A 

Sandra Regina Sales Universidade Federal Rural do 
Rio de Janeiro, Brasil 

Elba Siqueira Sa Barreto Fundacao Carlos Chagas. 
Brasil 

Manuela Terraseca Universidade do Porto, Portugal 

Robert Verhine Universidade Federal da Bahia, 

Brasil 

Antonio A. S. Zuin Universidade Federal de Sao 
Carlos, Brasil 



